Stochastic Coordinate Descent for Nonsmooth Convex Optimization

Stochastic coordinate descent, due to its practicality and efficiency, is increasingly popular in machine learning and signal processing communities as it has proven successful in several large-scale optimization problems , such as l1 regularized regression, Support Vector Machine, to name a few. In this paper, we consider a composite problem where the nonsmoothness has a general structure that is compatible with a coordinate partition, and we solve the nonsmooth optimization problem using a sequence of smooth approximations. In particular, we extend Nesterov’s estimate sequence technique by incorporating smooth approximation and coordinate randomization. By studying the effect of smooth approximation, we develop rules for selecting smooth approximations that not only guarantee the algorithm’s convergence but also provide better convergence rate than the subgradient black-box model. Specifically, we obtain the convergence rate of O ( 1 K ) for nonsmooth convex functions and O ( 1 K2 ) for strongly convex functions. The convergence analysis developed in this paper and the results, to the best of our knowledge, have not been shown previously for stochastic coordinate descent.